Black Hat Bash - Bash Scripting for Hackers and Pentesters (Early Access) (Dolev Farhi, Nick Aleks)

(%{size_request}), the size of the headers returned in bytes

(%{size_header}), and more.

#!/bin/bash

TARGET_URL="http://172.16.10.12"

ROBOTS_FILE="robots.txt"

1 while read -r line; do

2 path=$(echo "${line}" | awk -F'Disallow: ' '{print $2}')

3 if [[ -n "${path}" ]]; then

url="${TARGET_URL}${path}"

status_code=$(curl -s -o /dev/null -w "%{http_code}" "${url}")

echo "URL: ${url} returned a status code of: ${status_code}"

fi

4 done < <(curl -s "${TARGET_URL}/${ROBOTS_FILE}")

Listing 5-1

A bash script that reads robots.txt and checks individual paths

At 1 we read the output from the curl command at 4 line by

line. This command makes an HTTP GET request to

http://172.16.10.12/robots.txt. We then parse each line and grab the

second field (which is separated from the others by a space) to

extract the path and assign it to the path variable 2. We check that

the path variable length is greater than zero to ensure we were able

to properly parse it at 3. Then we create a url variable, which is a

string concatenated from the TARGET_URL variable plus each path

from robots.txt file, and make an HTTP request to the URL. We then

use the -w (write-out) variable %{http_code} to extract only the

status code from the response returned by the web server.

Try using other cURL variables in your own scripts. The full list

of variables can be found here at https://curl.se/docs/manpage.html

or by running the man curl command. You can download the

script shown in this section at https://github.com/dolevf/Black-Hat-

Bash/blob/master/ch05/curl_fetch_robots_txt.sh.

Brute-Forcing Directories with dirsearch

Dirsearch is a fast directory brute-forcing tool used to find

hidden paths and files on web servers. Written in Python by Mauro

Soria, dirsearch provides features such as built-in web directory

wordlists, bring-your-own-dictionary options, advanced response

filtering, and more. We’ll use it to try to identify additional attack

vectors and verify that Nikto hasn’t missed anything obvious.